Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "Add command line arg to allow no restart test runner on new test groups" #34478

Closed
wants to merge 1 commit into from

Conversation

gsnedders
Copy link
Member

Reverts #34133.

To quote #34474 (comment):

Previously:

 0:19.03 TEST_START: /css/cssom-view/scroll-behavior-smooth-navigation.html
 0:37.04 INFO Got timeout in harness
 0:37.04 DEBUG Got command: 'test_ended'
 0:37.04 DEBUG Unexpected count in this thread 1
 0:37.04 TEST_END: TIMEOUT, expected ERROR - TestRunner hit external timeout (this may indicate a hang)
 0:37.04 DEBUG new state: restarting
 0:37.04 DEBUG Dispatch restart_runner
 0:37.04 DEBUG Stopping WebDriver
 0:37.07 DEBUG OutputHandler.after_process_stop
 0:37.07 DEBUG Going to stop Safari
 0:37.14 DEBUG Stopping Safari 46683
 0:37.17 DEBUG ensure_runner_stopped
 0:37.17 DEBUG Stopping WebDriver
 0:37.17 DEBUG Going to stop Safari
 0:37.23 DEBUG waiting for runner process to end
 0:37.23 DEBUG After join
 0:37.23 DEBUG Runner process exited with code 0
 0:37.23 DEBUG TestRunnerManager cleanup
 0:37.06 WARNING Traceback (most recent call last):
  File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/executors/base.py", line 301, in run_test
    result = self.do_test(test)
  File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 459, in do_test
    success, data = WebDriverRun(self.logger,
  File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/executors/base.py", line 205, in run
    if self.protocol.is_alive():
  File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 385, in is_alive
    self.webdriver.send_session_command("GET", "window", timeout=2)
  File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/webdriver/webdriver/client.py", line 661, in send_session_command
    return self.send_command(method, url, body, timeout)
  File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/webdriver/webdriver/client.py", line 613, in send_command
    response = self.transport.send(
  File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py", line 234, in send
    response = self._request(method, uri, payload, headers, timeout=None)
  File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py", line 259, in _request
    response = self.connection.getresponse()
  File "Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1349, in getresponse
    response.begin()
  File "Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

 0:37.23 WARNING Command left in command_queue during cleanup: 'test_ended', (<wptrunner.wpttest.TestharnessTest /css/cssom-view/scroll-behavior-smooth-navigation.html>, (<wptrunner.wpttest.TestharnessResult INTERNAL-ERROR>, []))
 0:37.06 DEBUG Hanging up on WebDriver session
 0:37.06 INFO Closing logging queue
 0:37.06 INFO queue closed
 0:37.23 DEBUG new state: initializing
 0:37.23 DEBUG Dispatch init
 0:37.23 DEBUG Init called, starting browser and runner
 0:37.23 DEBUG Starting browser with settings {}
 0:37.23 DEBUG Starting WebDriver: /Applications/Safari Technology Preview.app/Contents/MacOS/safaridriver --port=58923
 0:37.24 DEBUG OutputHandler.after_process_start
 0:37.24 DEBUG Trying to connect to 127.0.0.1:58923
 0:37.74 DEBUG Connected to 127.0.0.1:58923
 0:37.74 DEBUG OutputHandler.start
 0:37.74 DEBUG _run complete
 0:37.74 INFO Starting runner
 0:37.75 DEBUG Test runner started
 0:38.96 DEBUG Got command: 'log'
 0:38.95 DEBUG Executor setup
 0:38.96 DEBUG Got command: 'log'
 0:38.95 DEBUG Connecting to WebDriver on URL: http://127.0.0.1:58923/
 0:40.26 DEBUG Got command: 'log'
 0:40.26 DEBUG Loading http://web-platform.test:8000/testharness_runner.html
 0:40.34 DEBUG Got command: 'init_succeeded'
 0:40.34 DEBUG new state: running
 0:40.34 DEBUG Dispatch run_test
 0:40.34 TEST_START: /css/cssom-view/window-screen-width.html
 0:40.34 DEBUG Got command: 'log'
 0:40.34 DEBUG Executor setup done
 0:40.63 DEBUG Got command: 'log'
 0:40.63 DEBUG Got async callback: complete
 0:40.71 DEBUG Got command: 'test_ended'
 0:40.71 TEST_END: Test OK. Subtests passed 3/3. Unexpected 0

After 62c3bc7 (#34133):

 0:31.30 TEST_START: /css/cssom-view/scroll-behavior-smooth-navigation.html
0:49.30 INFO Got timeout in harness
0:49.30 DEBUG Got command: 'test_ended'
0:49.30 DEBUG Unexpected count in this thread 1
0:49.30 TEST_END: TIMEOUT, expected ERROR - TestRunner hit external timeout (this may indicate a hang)
0:49.30 DEBUG new state: running
0:49.30 DEBUG Dispatch run_test
0:49.30 TEST_START: /css/cssom-view/window-screen-width.html
1:04.80 INFO Got timeout in harness
1:04.80 DEBUG Got command: 'test_ended'
1:04.80 DEBUG Unexpected count in this thread 2
1:04.80 TEST_END: TIMEOUT, expected OK - TestRunner hit external timeout (this may indicate a hang)

Thus we see a change in behaviour with external timeouts following #34133, reverting that.

I'm not sure what's wrong with the change in #34133, but this is completely reproducible locally when the test times out.

@gsnedders gsnedders requested review from jgraham and WeizhongX June 17, 2022 02:02
@wpt-pr-bot wpt-pr-bot added infra wptrunner The automated test runner, commonly called through ./wpt run labels Jun 17, 2022
@gsnedders
Copy link
Member Author

Scrap this, I've just realised where the bug is. Let's just fix it.

@gsnedders gsnedders closed this Jun 17, 2022
@gsnedders gsnedders deleted the revert-34133-no-restart-on-new-group branch June 17, 2022 02:06
@gsnedders
Copy link
Member Author

See #34479 for the actual fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infra wptrunner The automated test runner, commonly called through ./wpt run
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants